<!DOCTYPE html>
<html lang="en" data-color-mode="auto" data-light-theme="light" data-dark-theme="dark">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>G4F - Local Usage Guide</title>
    <link rel="apple-touch-icon" sizes="180x180" href="/dist/img/apple-touch-icon.png">
    <link rel="icon" type="image/png" sizes="32x32" href="/dist/img/favicon-32x32.png">
    <link rel="icon" type="image/png" sizes="16x16" href="/dist/img/favicon-16x16.png">
    <link rel="manifest" href="/dist/img/site.webmanifest">
    <link crossorigin="anonymous" media="all" rel="stylesheet" href="https://github.githubassets.com/assets/light-74231a1f3bbb.css" />
    <link crossorigin="anonymous" media="all" rel="stylesheet" href="https://github.githubassets.com/assets/dark-8a995f0bacd4.css" />
    <link crossorigin="anonymous" media="all" rel="stylesheet" href="https://github.githubassets.com/assets/primer-primitives-225433424a87.css" />
    <link crossorigin="anonymous" media="all" rel="stylesheet" href="https://github.githubassets.com/assets/primer-b8b91660c29d.css" />
    <link crossorigin="anonymous" media="all" rel="stylesheet" href="https://github.githubassets.com/assets/global-205098e9fedd.css" />
    <link crossorigin="anonymous" media="all" rel="stylesheet" href="https://github.githubassets.com/assets/code-177d21388df8.css" />
    <style>
        :root {
            --colour-1: #000000;
            --colour-2: #ccc;
            --colour-3: #e4d4ff;
            --colour-4: #f0f0f0;
            --colour-5: #181818;
            --colour-6: #242424;
            --accent: #8b3dff;
            --gradient: #1a1a1a;
            --background: #16101b;
            --size: 70vw;
            --top: 50%;
            --blur: 40px;
            --opacity: 0.6;
        }

        /* Body and text color */
        body {
            height: 100vh;
            margin: 0;
            padding: 0;
        }

        .hidden {
            display: none;
        }

        .container-lg {
            margin: 0 auto;
            padding: 8px;
        }

        @media only screen and (min-width: 40em) {
            .container-lg {
                max-width: 84%;
            }
        }
    </style>
</head>
<body>
    <article class="markdown-body entry-content container-lg" itemprop="text"><div class="markdown-heading"><h2 class="heading-element">G4F - Local Usage Guide</h2><a id="user-content-g4f---local-usage-guide" class="anchor" aria-label="Permalink: G4F - Local Usage Guide" href="#g4f---local-usage-guide"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="markdown-heading"><h3 class="heading-element">Table of Contents</h3><a id="user-content-table-of-contents" class="anchor" aria-label="Permalink: Table of Contents" href="#table-of-contents"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<ol>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#required-dependencies">Required Dependencies</a></li>
<li><a href="#basic-usage-example">Basic Usage Example</a></li>
<li><a href="#supported-models">Supported Models</a></li>
<li><a href="#performance-considerations">Performance Considerations</a></li>
<li><a href="#troubleshooting">Troubleshooting</a></li>
</ol>
<div class="markdown-heading"><h4 class="heading-element">Introduction</h4><a id="user-content-introduction" class="anchor" aria-label="Permalink: Introduction" href="#introduction"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>This guide explains how to use g4f to run language models locally. G4F (GPT4Free) allows you to interact with various language models on your local machine, providing a flexible and private solution for natural language processing tasks.</p>
<div class="markdown-heading"><h2 class="heading-element">Usage</h2><a id="user-content-usage" class="anchor" aria-label="Permalink: Usage" href="#usage"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="markdown-heading"><h4 class="heading-element">Local inference</h4><a id="user-content-local-inference" class="anchor" aria-label="Permalink: Local inference" href="#local-inference"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>How to use g4f to run language models locally</p>
<div class="markdown-heading"><h4 class="heading-element">Required dependencies</h4><a id="user-content-required-dependencies" class="anchor" aria-label="Permalink: Required dependencies" href="#required-dependencies"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p><strong>Make sure to install the required dependencies by running:</strong></p>
<div class="highlight highlight-source-shell"><pre>pip install g4f[local]</pre></div>
<p>or</p>
<div class="highlight highlight-source-shell"><pre>pip install -U gpt4all</pre></div>
<div class="markdown-heading"><h4 class="heading-element">Basic usage example</h4><a id="user-content-basic-usage-example" class="anchor" aria-label="Permalink: Basic usage example" href="#basic-usage-example"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">g4f</span>.<span class="pl-s1">local</span> <span class="pl-k">import</span> <span class="pl-v">LocalClient</span>

<span class="pl-s1">client</span>   <span class="pl-c1">=</span> <span class="pl-en">LocalClient</span>()
<span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">client</span>.<span class="pl-c1">chat</span>.<span class="pl-c1">completions</span>.<span class="pl-c1">create</span>(
    <span class="pl-s1">model</span>    <span class="pl-c1">=</span> <span class="pl-s">'orca-mini-3b'</span>,
    <span class="pl-s1">messages</span> <span class="pl-c1">=</span> [{<span class="pl-s">"role"</span>: <span class="pl-s">"user"</span>, <span class="pl-s">"content"</span>: <span class="pl-s">"hi"</span>}],
    <span class="pl-s1">stream</span>   <span class="pl-c1">=</span> <span class="pl-c1">True</span>
)

<span class="pl-k">for</span> <span class="pl-s1">token</span> <span class="pl-c1">in</span> <span class="pl-s1">response</span>:
    <span class="pl-en">print</span>(<span class="pl-s1">token</span>.<span class="pl-c1">choices</span>[<span class="pl-c1">0</span>].<span class="pl-c1">delta</span>.<span class="pl-c1">content</span> <span class="pl-c1">or</span> <span class="pl-s">""</span>)</pre></div>
<p>Upon first use, there will be a prompt asking you if you wish to download the model. If you respond with <code>y</code>, g4f will go ahead and download the model for you.</p>
<p>You can also manually place supported models into <code>./g4f/local/models/</code></p>
<p><strong>You can get a list of the current supported models by running:</strong></p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">g4f</span>.<span class="pl-s1">local</span> <span class="pl-k">import</span> <span class="pl-v">LocalClient</span>

<span class="pl-s1">client</span>   <span class="pl-c1">=</span> <span class="pl-en">LocalClient</span>()
<span class="pl-s1">client</span>.<span class="pl-c1">list_models</span>()</pre></div>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-ent">"mistral-7b"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>mistral-7b-openorca.gguf2.Q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>8<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;|im_start|&gt;user<span class="pl-cce">\n</span>%1&lt;|im_end|&gt;<span class="pl-cce">\n</span>&lt;|im_start|&gt;assistant<span class="pl-cce">\n</span><span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;|im_start|&gt;system<span class="pl-cce">\n</span>You are MistralOrca, a large language model trained by Alignment Lab AI. For multi-step problems, write out your reasoning for each step.<span class="pl-cce">\n</span>&lt;|im_end|&gt;<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"mistral-7b-instruct"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>mistral-7b-instruct-v0.1.Q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>8<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>[INST] %1 [/INST]<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"gpt4all-falcon"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>gpt4all-falcon-newbpe-q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>8<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>### Instruction:<span class="pl-cce">\n</span>%1<span class="pl-cce">\n</span>### Response:<span class="pl-cce">\n</span><span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"orca-2"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>orca-2-13b.Q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>16<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-ii">None,</span>
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"wizardlm-13b"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>wizardlm-13b-v1.2.Q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>16<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-ii">None,</span>
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"nous-hermes-llama2"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>nous-hermes-llama2-13b.Q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>16<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>### Instruction:<span class="pl-cce">\n</span>%1<span class="pl-cce">\n</span>### Response:<span class="pl-cce">\n</span><span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"gpt4all-13b-snoozy"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>gpt4all-13b-snoozy-q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>16<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-ii">None,</span>
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"mpt-7b-chat"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>mpt-7b-chat-newbpe-q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>8<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;|im_start|&gt;user<span class="pl-cce">\n</span>%1&lt;|im_end|&gt;<span class="pl-cce">\n</span>&lt;|im_start|&gt;assistant<span class="pl-cce">\n</span><span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;|im_start|&gt;system<span class="pl-cce">\n</span>- You are a helpful assistant chatbot trained by MosaicML.<span class="pl-cce">\n</span>- You answer questions.<span class="pl-cce">\n</span>- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.<span class="pl-cce">\n</span>- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.&lt;|im_end|&gt;<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"orca-mini-3b"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>orca-mini-3b-gguf2-q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>4<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>### User:<span class="pl-cce">\n</span>%1<span class="pl-cce">\n</span>### Response:<span class="pl-cce">\n</span><span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-s"><span class="pl-pds">"</span>### System:<span class="pl-cce">\n</span>You are an AI assistant that follows instruction extremely well. Help as much as you can.<span class="pl-cce">\n\n</span><span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"replit-code-3b"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>replit-code-v1_5-3b-newbpe-q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>4<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>%1<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"starcoder"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>starcoder-newbpe-q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>4<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>%1<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"rift-coder-7b"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>rift-coder-v0-7b-q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>8<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>%1<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"all-MiniLM-L6-v2"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>all-MiniLM-L6-v2-f16.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-ii">None,</span>
        <span class="pl-ent">"system"</span>: <span class="pl-ii">None</span>
    },
    <span class="pl-ent">"mistral-7b-german"</span>: {
        <span class="pl-ent">"path"</span>: <span class="pl-s"><span class="pl-pds">"</span>em_german_mistral_v01.Q4_0.gguf<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"ram"</span>: <span class="pl-s"><span class="pl-pds">"</span>8<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"prompt"</span>: <span class="pl-s"><span class="pl-pds">"</span>USER: %1 ASSISTANT: <span class="pl-pds">"</span></span>,
        <span class="pl-ent">"system"</span>: <span class="pl-s"><span class="pl-pds">"</span>Du bist ein hilfreicher Assistent. <span class="pl-pds">"</span></span>
    }
}</pre></div>
<div class="markdown-heading"><h4 class="heading-element">Performance Considerations</h4><a id="user-content-performance-considerations" class="anchor" aria-label="Permalink: Performance Considerations" href="#performance-considerations"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p><strong>When running language models locally, consider the following:</strong></p>
<ul>
<li>RAM requirements vary by model size (see the 'ram' field in the model list).</li>
<li>CPU/GPU capabilities affect inference speed.</li>
<li>Disk space is needed to store the model files.</li>
</ul>
<div class="markdown-heading"><h4 class="heading-element">Troubleshooting</h4><a id="user-content-troubleshooting" class="anchor" aria-label="Permalink: Troubleshooting" href="#troubleshooting"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p><strong>Common issues and solutions:</strong></p>
<ol>
<li>
<strong>Model download fails</strong>: Check your internet connection and try again.</li>
<li>
<strong>Out of memory error</strong>: Choose a smaller model or increase your system's RAM.</li>
<li>
<strong>Slow inference</strong>: Consider using a GPU or a more powerful CPU.</li>
</ol>
<hr>
<p><a href="/docs/">Return to Documentation</a></p>
</article>
</body>
</html>